Method for Improved Handling of Texture Data For Texturing and Other Image Processing Tasks

ABSTRACT

An imagery processing system that combines MIP level filtering with spatial filtering when rendering images. Filtering can be performed in an order that optimizes memory accesses during the rendering process.

CROSS-REFERENCES TO PRIORITY AND RELATED APPLICATIONS

This application claims the priority benefit of U.S. Provisional PatentApplication No. 62/968,120 filed Jan. 30, 2020 and entitled “Method forCombining MIP Level Filtering with Spatial Filtering During Texturing”and claims the priority benefit of U.S. Non-Provisional patentapplication Ser. No. 17/163, 325 filed Jan. 29, 2021 and entitled“Method For Improved Handling Of Texture Data For Texturing And OtherImage Processing Tasks”. These applications are hereby incorporated byreference in their entirety as though fully set forth herein.

FIELD

The present disclosure generally relates to methods and systems forprocessing data used for rendering computer-generated images. Thedisclosure, in part, relates more particularly to apparatus andtechniques for exploiting multi-threaded computing environments topre-calculate filters to be applied for efficient rendering.

BACKGROUND

Many industries generate or use computer-animated features within animage or series of images. To render lifelike images, one or moretexture tiles might be applied to a defined surface of a model. Multipletiles can be coalesced together to account for color and materialproperties of what the surface represents, such as roughness topographyand reflectivity. Texture engines read one or more tiles from storage,calculate one or more filter values to apply to the tile, and apply theone or more filters to the tile to account for lighting conditions of ascene where a ray trace extends from an audience view of the scene tothe surface. This serializing technique of reading a tile, calculating afilter value, and applying the filter for many tiles is resourceintensive, which results in extended render times or constrains thenumber of texture tiles read (effectively the size of a pixel region toapply a filter) or the number of filters calculated.

SUMMARY

A rendering system for generating rendered imagery, such as videosequences or still images, from computer-described objects can includeapplying a filter, calculating a fitted filter based on a determinedimpact of applying a filter across a range of tile vertices, accessingone or more tiles of one or more MIP levels determined to correspond toa pixel region selected for filtering, and applying the at least onefitted filter to the accessed one or more tiles. One or more fittedtwo-dimensional (2D) filters might be used, having various shapes. Insome embodiments, a filter might be implemented as a filter kernel.

The fitted filter might be a first fitted filter and a plurality offitted filters are calculated and accumulated in a cache beforedetermining the one or more tiles of one or more MIP levels to access.In some embodiments, a bilinear filter might be collapsed or merged toallow for point sampling and correspondingly less computation andcaching requirements.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tolimit the scope of the claimed subject matter. A more extensivepresentation of features, details, utilities, and advantages of thesurface computation method, as defined in the claims, is provided in thefollowing written description of various embodiments of the disclosureand illustrated in the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example 100 of where texturing might be done.

FIG. 2 illustrates an example of a texture processing system 200 asmight be used for embodiments of texture processing.

FIG. 3 is a block diagram illustrating an embodiment of a system inwhich a texture engine of a computing system may process a sampled pixelregion by determining MIP maps of a tile and applying a precomputedfilter.

FIG. 4 is a block diagram depicting an embodiment of the processillustrated in FIG. 3 , including applying a filter to a series of tileswithin and across MIP map levels.

FIG. 5 is a flow chart depicting a process for applying a texture to atile.

FIG. 6 illustrates an example visual content generation system as mightbe used to generate imagery in the form of still images and/or videosequences of images, according to various embodiments.

FIG. 7 is a block diagram illustrating an example computer system uponwhich computer systems of the systems illustrated in FIGS. 3 and 6 maybe implemented.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

The present disclosure describes techniques for efficiently processingdata to generate imagery therefrom, wherein imagery might be one or moreimages representing real live action scenes, virtual scenes, orcombinations thereof.

In some examples, objects in a scene can be represented in computermemory by a data structure defining a mathematical representation of atwo-dimensional (2D) or three-dimensional (3D) physical item orcharacter (e.g., a human, an animal, a fanciful being, etc.). Togenerate an image, a renderer, or other computer system, might read inthat data structure, determine a position of a camera view point in avirtual space, the location and orientation of a camera view plane inthe virtual space, and then use various techniques to determine whatcolors to assign to pixels of the camera view plane based on objectlocation, light effects, etc. on the objects positioned in that virtualspace. One approach to determining a pixel color for a pixel of thecamera view plane is by ray tracing, wherein a computer processdetermines what would occur as a ray of light follows a path,interacting with objects in the scene, and intersects the pixel of thecamera view plane and the camera view point, possibly simulating effectsof the light ray's encounters with virtual objects. A “ray trace” mightrefer to such a path of light, which might be defined as beginning atthe camera view point and ending at some object or beginning at somelight source or object and ending at the camera view point.

The representation of an object might be stored as a mesh, wherein amesh might be defined by a set of vertices and a set of edges betweenvertices, therein defining faces of the mesh and collectively forming amanifold of a boundary for the object. A texture for coloring thatboundary might be provided. Faces of the mesh might be triangles orother polygons defining a shape of a polyhedral object representing theobject for display or computer graphics modeling processes. In someexamples, a scene can be represented as data structures referring to avirtual workspace for editing and/or rendering a 3D project. In someexamples, an interface might be computer hardware or software designedto communicate information between hardware devices, between softwareprograms, between devices and programs, or between a device and a user.Often, fine details are desired and an ability to generate imagery withthose fine details without requiring tedious user input would allow forgreater facility with such imagery.

In some examples, a scene defining an arrangement of virtual objects,light sources, effects, etc. in a 2D or 3D space can be represented incomputer memory be data defining geometric models. A digital artist orother creator might use a computer interface to specify details of ascene. This might be in the form of a virtual workspace for editingand/or rendering a 3D project. Some objects might be specified by asmall number of parameters, such as primvars or geometric primitivevariables, such as points, lines, line segments, planes, circles,ellipses, triangles or other polygons, and/or spline curves. In someexamples, primvars alternatively or additionally include spheres, cubes,toroids, cylinders, pyramids, and/or polygon meshes.

Portions of the scene might be described procedurally based on someprimvars. For example, a spherical object might be represented in memoryby primvars such as a center position, a radius, and a surface color,and a computer image generation system can generate imagery of thatspherical object procedurally from the stored primvars.

In some examples, an “interface” refers to computer hardware or softwaredesigned to communicate information between hardware devices, betweensoftware programs, between devices and programs, or between a device anda user. In some examples, “communication” refers to a transmissionand/or exchange of information. In other examples, a “display device”(also referred to as a “monitor” or “screen”) refers to a device capableof representing information (e.g., words, numbers, drawings, images,etc.) visually (e.g., by a cathode-ray tube, liquid crystal display,light emitting diode, projection surface, etc.) that provides visualoutput from an image or video output device (e.g., computer, cable box,video camera, etc.). In some uses, methods and apparatus are used togenerate the data of imagery, such as images in the form of arrays ofpixel colors, with or without depth information, and/or a sequence ofimages forming frames of a video sequence.

In many cases, an object can be represented by a mesh that defines aclosed and continuous surface. For texture mapping coloration onto thatmesh for the object, a pre-computed set of textures might be used. Thatset of pre-computed textures might be computed at various resolutions.

“MIP mapping, or “mipmapping,” is a process of pre-computing optimizedsequences of images, at varying resolutions, representing the sameimage. A “MIP map” or “mipmap” may refer to a data structure thatresults from mipmapping, comprising a plurality of MIP levels. As aspecific example, a mipmap might comprise pixel arrays of a texture forthe exterior of a virtual building, such as a first MIP level that is a128×128 pixel array, a second MIP level that is a 64×64 pixel array,etc., possibly down to a 1×1 pixel array with each MIP level havingone-fourth the number of pixels as the next highest resolution MIPlevel. In other variations, the resolution steps between MIP levelsmight be different and might vary. The mipmap might be useful whengenerating a scene that includes the building. When the building isclose to the camera, a higher resolution MIP level might be usedcompared with when the building is far from the camera and is going toaffect many fewer pixels of the resulting image.

Images might be stored in a format wherein the image is subdivided intological or physical blocks referred to as “tiles”, such as square orother two-dimensional geometric shapes comprising four vertices.

As one skilled in the art will appreciate in light of this disclosure,certain embodiments can be capable of achieving certain advantages byutilizing parallel processing of computing filters and reading tiles,including some or all of the following: (1) decreasing processing timeto render a compute-generated image for a specified image resolution,(2) improving resolution of a rendered image (e.g., reading additionaltiles or calculating additional filters) for a specified processingtime, (3) increasing scale of a filter (e.g. size of sampled pixels)that is applied within a specified processing time; and (4) improvingapplication of one or more filters by pre-calculating filter functionsindependent of tile read/write operations, such that a cache ofpre-calculated filters may be applied to an unconstrained quantity oftiles.

In general, a texturing process might involve determining, computing, orreceiving an indication of a filter region and computing and storing,returning, or transmitting a resultant output texturing data value. Thatoutput texturing data value could represent a color value or it couldrepresent something other than color. An output texturing value might bea scalar value or a vector value. The filter region might correspond toa set of points or positions that correspond to a view from a pixel inscreen space, a voxel, or something else. Often determining an outputtexturing data value is part of a process wherein many output texturingdata values are determined. A texture dataset can be a file, stream,and/or computed set of values, that correspond to an array of texturevalues, or texels. A texture dataset can be read from some memory, orover a network, or perhaps computed on the fly from a set of textureparameters.

In a very specific example, in an image rendering of an object intopixel color values in a screen space, a portion of a surface of theobject in a virtual scene space that covers a pixel is determined, thatportion might be mapped onto a UV map of the surface of the object, atexture dataset is read to determine the values for that portion of theUV map, a filter kernel is computed that represents relative weights oftexels of a texture dataset, and weighted texels from the texturedataset are aggregated, such as by summing, to arrive at the resultantoutput texturing data value. This process can be performed using severaltextures if multiple texture datasets are to be used to influence theresultant output texturing data value. This process can be repeated overseveral pixels, such as where an entire view of the object from a cameraposition is to be computed. A filter kernel might correspond to aweighted sum of neighbors around a particular location and mightcorrespond to interpolating results between two resolution levels.

In one example, the filter region is determined by projecting a disc (orother shape) onto a surface at some orientation and distance. Theprojection might be a slice of a cone, such as an ellipse. Filteringmight be applied at the intersection of the projected shape and thesurface to approximate that intersection, possibly adjusting to avoidaliasing. The filter region might be a small area covering multipletexels as necessary.

A texture dataset might include multiple prefiltered resolution levelsof detail. A multilevel resolution texture dataset might comprise atexture image at a first level, a texture image at a second level thatis a lower resolution than the first level texture image, a third level,and so on. It might be that the resolution drops by some factor from onelevel to the next and drops in each dimension, as in a MIP-map datastructure. That resolution could be a factor of two or some otherfactor. The resolution might drop by the same factor in each dimension,as in a MIP-map data structure, or by different factors for differentdimensions, such as a RIP-map data structure. In some variations,wavelet data structures might be used or some other encoding whereinhigher resolution levels build on lower resolution levels. However, atexture dataset with a single level of detail could also be used. Atexture dataset might comprise 2D arrays of values for a surfacetexture, but could comprise a different number of dimensions, such as a3D array of values for a volume texture.

In many instances, there are multiple textures that need to be takeninto account and they might not be aggregated into a single texturedataset. For example, a rendering process might rely on one texturedataset representing a base color and another texture datasetrepresenting a “shininess” component. Other examples include a roughnesstexture, a “rust” texture, etc. A moderately complex scene might involveobjects that are textured with dozens or hundreds of textures beingsampled for each virtual hit point. In some cases, a texture datasetcomprises a large number of texels and only a small number of them areneeded for sampling. It may be that the relative positions of the texelsbeing sampled are determinable and are mostly the same for each texturedataset being considered and it may be that the texels of a texturedataset are organized into tiles (such as 32×32 arrays of values) and itmay be that the same tile locations with each texture dataset are usedand not used, each typically having different tile values and kernelweights to sample those tile might be the same.

In some variations, post-aggregation filtering is applied to theresulting output texturing data values. In some variations, thepost-aggregation filtering could be applied to the filter kernel that isused before the computation of the output texturing data values. Forexample, a transform operation and/or a blur operation that might beapplied in compositing might be applied to the filter kernel in thetexturing process. Other examples include interpolation across a gridand filtering between MIP levels).

As explained herein, the filter weights of the filter kernel might beused over and over and thus computing them and storing them for lateruse can save on computational efforts. Also, for a given filter region,the tile selection for texture datasets might be the same over multipletexture datasets, so computing them and storing them for later use cansave on computational efforts. Computation of the filtering kernel canbe separated from step of texture dataset access. By caching kernelweights in a sparse grid, for example, for reuse when the filter regionis same might reduce a need to recompute filter weights, tiles needed,etc.

A texturing process or system might schedule and store what tiles areneeded for a particular sampling operation, which might allow theprocess or system to batch requests for tiles against a file load. Thiscan reduce thread contention. If the different texture datasets beingsampled have similar MIP level layouts, the process or system canrequest the appropriate tiles from a cache or a source such that therequests are more ordered. In a specific embodiment, the requests areordered to result in reading from a file in a linear fashion.

FIG. 1 illustrates an example 100 of where texturing might be done. Thevisual elements depicted there might be represented by data structuresstored in memory or storable in memory that can be processed using stepsdescribed herein. A texturing process might be executed to determine acolor value or other value for a pixel 102 (or an array of pixels) thatis to be driven by one or more textures associated with a surface 104 ina scene. In some rendering processes, a ray is drawn from a pixel intothe scene until it intersects some object. In other processes, texturingproceeds as if a cone 103 (or some other expanding volume) is emittedfrom pixel 102 until it intersects surface 104. An intersection of cone103 and surface 104 might be defined as a surface pattern 106, whichmight be an ellipse, a quadrangle, or other shape. Perhaps points onsurface 106 correspond to coordinates on a texture. In this example,texturing involves a plurality of textures 114 and within a texture114(1), a subset 118(1) of texels of texture 114(1), are used todetermine values for pixel 102 and other texels are not. Depending onthe bounds of the texels and the shape of surface pattern 106, someweighting function might be applied to the texels that are in subset118(1).

In a more general case, the texturing operation determines a sample toreturn for a sample region that might be a region represented by apixel's extent on a surface that is in turn mapped to a UV map 110 fortextures, or simply some filter region representation comprisingcomputer-readable data corresponding to a sample region of a virtualmapping to be sampled. Textures and kernels can then be computed forthat filter region. In some embodiments, the filter region is specifiedas the centroid of an ellipse, the direction of at least one axis, andthe magnitudes of the major and minor axes. The filter region might bederived by mapping a surface in a virtual space onto UV map 110, but itmight be derived from some other source 108 and specified in a requestfor a sample.

In a particular process, the operation might comprise determining afirst texture dataset, determining a filter kernel for the filter regionrepresentation, the filter kernel corresponding to the sample region,wherein the filter kernel comprises one or more kernel weights over thesample region, storing the filter kernel, determining a subset of thefirst texture dataset upon which the filter kernel is to be applied,storing a schedule of the subset, wherein positions of texels within thefirst texture dataset are determinable from the schedule, applying theschedule and the filter kernel to the first texture dataset to determinea first texture contribution for the sample region, determining a secondtexture dataset, applying the schedule and the filter kernel to thesecond texture dataset to determine a second texture contribution forthe sample region, and combining the first texture contribution and thesecond texture contribution to determine an overall accumulated outputfor the sample region as the sample. Methods herein that describe usingsurface patterns could be used for other filter region representations.

In some embodiments, textures might be stored such that the firsttexture dataset and the second texture dataset are part of some largerdata structure. Those textures might be for different channels withinthat texture data structure, such as two or more color value channels,an alpha channel, or some other arrangement. A texture dataset might beone or more of a plurality of texture components or channels, perhapsless than all available components/channels.

In one approach, a texturing processor will compute a shape of cone 103based on pixel position, distance, etc. and compute surface pattern 106based on pixel position, distance and shape of surface 104 atintersecting locations. The texturing processor can compute which texelsare needed to compute a texture for pixel 102 based on, perhaps, theshape of surface 104 and a mapping (such as a UV mapping) of texels froma texture 114(1) to surface pattern 106. The texturing processor canalso compute a filter to use for pixel 102 and texture 114(1) thatcorresponds to the weights of the texels. For example, where thetexturing processor computes an overlap of surface pattern 106 andtexels of subset 118(1), some portions of some texels might be outsidethe pattern and thus assigned a weight of less than 100% the weightassigned to other texels.

Where there are multiple textures that are combined for depicting asurface or for computing surface values, the texturing processor couldcompute, for each pixel, a surface pattern, and from there a filterpattern, and then retrieve a texture dataset to determine texel valuesto combine and repeat this process for each texture. That can requireconsiderable computational effort.

For more efficient texture processing, the texturing processor can storeor cache a filter and can store or cache a texture schedule thatindicates which texels of a texture are used. In the example illustratedin FIG. 1 , there are multiple textures 114 that are combined fordetermining values within surface pattern 106 and if the textures have acommon coordinate system, the locations of the texels used in onetexture might be the same as the locations of the texels used from othertextures. If a schedule of texels is stored, it might need be computedonly once for a given surface pattern, saving computational effort.Furthermore, if textures are stored in data structures that areretrieved from memory using multiple memory accesses, a memory requestcan be for less than the entirety of a texture dataset, thus savingmemory access bandwidth. In such cases, the texturing processor mightread only those texels that are represented by a schedule of texels anduse that schedule for several texture datasets. Additionally, if thetextures are stored as multiple levels of MIP-MAP data, the schedule oftexels might indicate what MIP-MAP level is needed, further savingmemory accesses as the texturing processor can make smaller memoryrequests, just for the levels that are needed.

Further efficiencies might be available by storing or caching thefilters and texel subsets from pixel to pixel.

FIG. 2 illustrates an example of a texture processing system 200 asmight be used for embodiments of texture processing. In that example, arenderer 202 connects to a filter determiner 204 and ascheduler/retriever 206. In an operation for determining a pixel value,such as a pixel color value, renderer 202 might determine a set oftexels from one texture, apply filter weights to those texels, determinea set of texels from another texture, apply the filter weights to thosetexels and sum contributions (or perform some other operation) todetermine a pixel value over a plurality of textures. To determine afilter to use to weight texels of a texture, renderer 202 might make arequest of filter determiner 204, which might read in a filter 212 froma filter weights array cache 210 or compute filter 212 and store it infilter weights array cache 210 while providing it to renderer 202. Inother variations, renderer 202 computes filter 212 and provides it tofilter determiner 204 which simply acts as a cache.

With a filter in hand, or before, renderer 202 cam make a request fortexture tiles from texture datasets to scheduler/retriever 206, whichcan obtain those tiles by reading from a tile schedule cache 216 todetermine which tiles to request from a texture store 218. In a casewhere there might be several dozen textures to combine, renderer 202might receive the same filter weights array for each of the textures andthe texels received from each of the textures might be the texels fromthe same positions, per the stored schedule. Where many textures areprocessed, and/or high-resolution renders are performed, such cachingand such reusing can save considerable computational effort andconsiderable bandwidth.

FIG. 3 is a diagram of a data flow through a system 300 when the system300 is performing a process 400 (see FIG. 4 ) that calculates a filtervalue for a computer-generated image. A user may select a pixel region302 for applying one or more filters of a plurality of filters 306. Apixel region 302 may be sampled from an image file accessible to thetexture engine 304. In one example, a filtering service operating inconnection with the system 300 provides a web service applicationprogramming interface to which web service application programminginterface (API) calls can be submitted to the texture engine 304 forprocessing an indicated pixel region 302.

A pixel region 302 may be a region within a ray trace extending from alight source indicated within a graphical environment, reflecting off asurface of a modeled object, and extending to a viewing position of thegraphical environment associated with an audience's perspective of thereflected surface. The process 400 (see FIG. 4 ) uses parallel filteringand tile read/write (input/output) to apply one or more filters to oneor more tiles associated with a surface of a modeled object. The system300 may be configured to decouple filtering and tile read/write, thusimproving real-time processing and/or processor efficiency.

The system 300 is shown including a texture engine 304, a repository fora plurality of filters 306, and a repository for a plurality of MIP maps312 corresponding to a tile for a texture to be applied to a surfacewithin a pixel region 302. The texture engine 304 may be part of one ormore rendering or animation pipelines. The texture engine 304 may beconfigurable by at least one client computing device operated by atleast one human artist. The texture engine 304 may be implemented bysoftware executing on a computer system (e.g., a computer system 600illustrated in FIG. 6 ). The software may include Houdini™) softwaretools developed by SideFx, Maya™) software tools developed by AutodeskInc., and the like. The software might also include a proprietaryrenderer and/or a commercially-available renderer with custom plug-insor components.

The texture engine 304 is configured to receive a pixel region 302, oneor more filters 306, and one or more MIP maps 312 as input. A pixelregion 302 may indicate reflected light from a surface, such asdetermined by a ray trace. A filter of a plurality of filters 306 mayinclude a function including Gaussian, Lanczos, Mitchell,Blackman-Harris, and the like. A function of a filter may be parametricto minimize error or aliasing when applied to a texture MIP map. Afunction may be separable to reduce multiplication operations when thetexture engine 304 computes applying a filter function. A function mayalso be an arbitrary non-separable function. A function might be abi-cubic function, or a noon-symmetric 2D Gaussian filter. A functionmight be separable in that it can be applied in one dimension and thenin the other dimension.

In an embodiment, a texture engine 304 accesses a filter function from afilter repository storing a plurality of filters 306. A texture engine304 may be capable of multiple thread processing such that the textureengine 304 may make multiple calculations synchronously. In anembodiment, a texture engine 304 may determine a filter to apply toincrease audience perceived sharpness of a low-resolution texturedesigned for an audience to view when the sampled pixel regioncorresponds to a closer perceived distance than a viewing position fromwhich the sampled pixel region represents. In an embodiment, a textureengine 304 may determine a filter to apply to decrease audienceperceived sharpness of a high-resolution texture designed for anaudience to view when the sampled pixel region corresponds to a fartherperceived distance than a viewing position from which the sampled pixelregion represents.

Based on a filter selection, such as specified via an interface of atexture engine 304 by a user, the texture engine 304 may perform a fitstep or analysis of an impact of applying the selected filter across adesignated range of vertices for the sampled pixel region 302.

For example, the texture engine 304 could combine multiple filters intoa combined filter and use the combined filter, which would improveefficiency over running each filter separately and can improve accuracyby not accumulating errors values over multiple filter steps.

Based on the results of the fit step and using the selected filterfunction, a texture engine 304 may calculate a fitted function 308. Afitted function 308 may represent fitting two or more functionparameters together, allowing the texture engine 304 to interpolatebetween any neighboring MIP levels. A fitted function 308 then avoidshard-coded constants for a specific sized MIP level. A desired primaryfilter might correspond to a projection of a desired sample onto arectilinear grid, usually interpreted as a slice of a conic section,yielding an ellipse—although other projections are possible. Filters,such as from a bilinear sample or similar to approximate sub-pixelaccuracy, might be layered on top of that to account for aliasing of therectilinear grid, or other effects. Filter samples might be laterinterpolated further down a pipeline on vertices of a model that can beprojected backward and modeled. By fitting multiple filters, the textureengine 304 can combine multiple filters into a new and possiblyarbitrary filter that can be cached and reused, thus avoiding the needto recompute the fitting.

A fitted function 308 may be aligned with a filter shape, such asapplying a Gaussian for an elliptical, spherical, or slender filter fora tile. Alignment may be processed to account for possible rotations ofa filter (e.g., diagonal) as processing may apply to a two-dimensionalrange of vertices. A texture engine 304 may add a calculated fit filterto a filter cache 310. A texture engine may calculate additional fitfilters, such as an initialization step or while waiting for tiles ofMIP map levels 314 to be read into a buffer by other thread processes ofthe texture engine 304.

A pixel region to be filtered may be received by a texture engine 304 inan API call request. The API call might be made by some other processthat is responding to a displacement or other action that desires todetermine surface properties (e.g., colors, light transport behavior,etc.). The request may be in the form of a shading language call, suchas a RenderMan Shading Language (RSL) call, a Houdini VEX ShadingLanguage (VEX) call, a Gelato Shading Language call, an Open ShadingLanguage (OSL) call, an OpenGL Architecture Review Board (ARB) call, anOpenGL (GLSL) call, an Nvidia™ Cg call, a DirectX Shader AssemblyLanguage call, an Adobe Pixel Bender call, an Adobe Graphics AssemblyLanguage call, a PlayStation Shader Language (PSSL) call, an Apply™Metal Shading Language (Metal) call, or the like.

An initialization step for processing one or more tiles may includeverifying that the texture engine 304 has sufficient buffers allocatedbased on a grid size of the sampled pixel region 302. In an embodiment,an individual core of a plurality of cores in a computer system 600 maybe tasked with processing an individual grid, such that the computersystem 600 may process multiple grids synchronously as each coresynchronously processes a different grid. An initialization step may notbe repeated when a plurality of texture requests corresponds a regionwithin the same grid. For example, if a texture request for a surface ofa modeled object is a painted car, then a texture request for a regionof the car can share the same initialization step for filtering thecolor of the car, flakiness of the paint, glossiness of the paint,severity of rust, etc.

Based on an indicated pixel region 302, a texture engine 304 may read atexture dataset from storage. A texture dataset may indicate one or moreMIP map levels 314 to access and obtain corresponding tiles in arepository for MIP maps 312. When a view represented in a sampled pixelregion 302 would correspond to a resolution between neighboring MIP maplevels, a texture engine 304 may interpolate between the two levels toachieve an intermediate MIP map 316.

A texture engine 304 may perform an accumulation once all tilescorresponding to the sampled pixel region 302 are retrieved. A textureengine 304 may iterate a cache of fitted filters 310 over the obtainedtiles. An iteration may include loading a tile into memory, performingone or more Streaming SIMD Extensions (SSE) operations to the tile basedon one or more fitted filters, and returning an output textured value316. Utilizing SSE operations in this manner facilitates a relativelyquick multiple/add calculation compared to calculating a function orfitted filter to apply to a tile as the tile is loaded into memory.

FIG. 4 is a block diagram illustrating a process 400 of how a series ofMIP map levels may be synchronously processed with a fitted filter. Fora given surface 402 comprising a plurality of tiles, a tile 404 may beselected for processing, such as based on a selected pixel region 302 asdescribed above.

To apply a filter to a grid, a corresponding range of vertices isdivided by the texture engine into pieces. In an embodiment, each pieceis available to a thread for processing. A texture engine, such astexture engine 304, may determine identifiers of a texture to beretrieved from a thread contribution list. Identifiers may indicateproperties of a virtual material corresponding to a surface of an objectrepresented in a sampled pixel region 302, such as color andreflectivity.

A texture engine, such as texture engine 304, may determine a sub regionto apply a filter. A subregion may be a single point, also implying thatfor a single point there are four surrounding vertices of the pointwithin a given MIP map level. When a texture engine determines that aresolution corresponding to a perceived distance from a camera positionto a surface including the subregion corresponds between two MIP maplevels, at least the four surrounding vertices of the subregion pointfor a higher level 406 and a lower level 410 may be obtained by thetexture engine. The texture engine may determine which additional tilesto access for a selected region and subregion to include straddledtiles, such as for tile 408.

As a fitted filter may be pre-calculated for a grid, applying the filterto tiles 406, 408, and 410 may occur synchronously.

FIG. 5 is a flowchart of the process 500 that may be executed by thesystem 300 of FIG. 3 and used to apply one or more filters to a tile.Referring to FIG. 5 , a texture engine receives 502 a request to apply afilter to a sampled region. As described above, a request may include anAPI call and may be formatted in a shading language. A texture enginebifurcates the work flow process.

When the texture engine receives 504 a filter selection, the textureengine retrieves the selected filter function from a repository offilter functions. Then the texture engine determines the impact ofapplying the filter to the sampled region. Based on the impact ofapplying the filter, the texture engine determines 508 a fitted filterfor the sampled region. A fitted filter may comprise two functionparameters such that it is possible for the texture engine tore-interpolate between any neighboring MIP levels. In computing a fittedfilter, a texture engine may align a selected filter function to accountfor any rotation of a region to which the selected filter is to beapplied. The texture engine then adds 510 the fitted filter to a filtercache.

The texture engine determines 512 MIP map levels to obtain from a MIPmap repository. The texture engine may obtain MIP maps for neighboringlevels. The texture engine determines 514 a subregion of the sampledregion to apply one or more filters. The subregion may be a point,indicating that a range of vertices, such as for a particular patch, bedivided into pieces at least corresponding to a range of four verticesimmediately surrounding the point. The texture engine loads 516 thecorresponding tile associated with the subregion into a tile cache.

In some embodiments, the texture engine can perform an offline processthat performs an analysis step of determining effect on an image andfitting to interpolation parameters, which can allow for the use of anappropriate MIP map level. The texture engine might receive an inputrequesting some texture operation. The texture engine might allocatesufficient memory to perform the requested operation and then perform astep of dispatching a process to compute filter regions, perhaps in amultithreaded method. A range of vertices might be treated as a patchthat is processed by a thread of a plurality of threads.

In processing a patch, a thread might compute, for each vertex in thepatch, which tile IDs are needed for that vertex. Then, the thread mightcompute what subregions are being considered and record kernel values.The texture image might wait for the threads to finish and then cachethe tiles. While the input/output processes are occurring, threads mightwork on other processing. When processing a grid, one thread might beprocessing one grid to process, but that thread can also evaluate ashading function and the texture engine can keep a local cache ofshading values. Filter weights might be computed by combination ofdifferent filters. By optimizing the thread process, texture processingcan be performed using less computational effort.

Caching and overlapping allows for computation to be provided for otherpurposes. Combining filters can also save on computational effort. Someof the filters might be parametric functions.

In step 518, the texture engine dispatches the filter cache to beapplied to accumulated tiles. The texture engine then applies 520 thefitted filters to the tiles in the tile cache. An iteration of applyinga fitted filter to a tile may include loading a tile into memory,performing one or more Streaming SIMD Extensions (SSE) operations to thetile based on one or more fitted filters, and returning an outputtextured value as described above.

Note that if a threshold of tiles has accumulated in the tile cachecorresponding to the sampled region, thread processes currently assignedto accumulating tiles (steps 512-@f3@16) may be reassigned by a loadbalancer to contribute to thread processes for calculating one or morefitted filters (steps 504-@f3@10) corresponding to the sampled region oranother sampled region. Similarly, if a threshold of fitted filters haveaccumulated in the filter cache corresponding to the sampled region,thread processes currently assigned to calculating filters (steps504-@f3@10) may be reassigned by a load balancer to contribute to threadprocesses for accumulating tiles corresponding to the sampled region oranother sampled region.

For example, FIG. 6 illustrates the example visual content generationsystem 600 as might be used to generate imagery in the form of stillimages and/or video sequences of images. Visual content generationsystem 600 might generate imagery of live action scenes, computergenerated scenes, or a combination thereof. In a practical system, usersare provided with tools that allow them to specify, at high levels andlow levels where necessary, what is to go into that imagery. Forexample, a user might be an animation artist and might use visualcontent generation system 600 to capture interaction between two humanactors performing live on a sound stage and replace one of the humanactors with a computer-generated anthropomorphic non-human being thatbehaves in ways that mimic the replaced human actor's movements andmannerisms, and then add in a third computer-generated character andbackground scene elements that are computer-generated, all in order totell a desired story or generate desired imagery.

Still images that are output by visual content generation system 600might be represented in computer memory as pixel arrays, such as atwo-dimensional array of pixel color values, each associated with apixel having a position in a two-dimensional image array. Pixel colorvalues might be represented by three or more (or fewer) color values perpixel, such as a red value, a green value, and a blue value (e.g., inRGB format). Dimensions of such a two-dimensional array of pixel colorvalues might correspond to a preferred and/or standard display scheme,such as 1920-pixel columns by 1280-pixel rows or 4096-pixel columns by2160-pixel rows, or some other resolution. Images might or might not bestored in a certain structured format, but either way, a desired imagemay be represented as a two-dimensional array of pixel color values. Inanother variation, images are represented by a pair of stereo images forthree-dimensional presentations and in other variations, an imageoutput, or a portion thereof, might represent three-dimensional imageryinstead of just two-dimensional views. In yet other embodiments, pixelvalues are data structures and a pixel value can be associated with apixel and can be a scalar value, a vector, or another data structureassociated with a corresponding pixel. That pixel value might includecolor values, or not, and might include depth values, alpha values,weight values, object identifiers or other pixel value components.

A stored video sequence might include a plurality of images such as thestill images described above, but where each image of the plurality ofimages has a place in a timing sequence and the stored video sequence isarranged so that when each image is displayed in order, at a timeindicated by the timing sequence, the display presents what appears tobe moving and/or changing imagery. In one representation, each image ofthe plurality of images is a video frame having a specified frame numberthat corresponds to an amount of time that would elapse from when avideo sequence begins playing until that specified frame is displayed. Aframe rate might be used to describe how many frames of the stored videosequence are displayed per unit time. Example video sequences mightinclude 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other framerates. In some embodiments, frames are interlaced or otherwise presentedfor display, but for clarity of description, in some examples, it isassumed that a video frame has one specified display time, but othervariations might be contemplated.

One method of creating a video sequence is to simply use a video camerato record a live action scene, i.e., events that physically occur andcan be recorded by a video camera. The events being recorded can beevents to be interpreted as viewed (such as seeing two human actors talkto each other) and/or can include events to be interpreted differentlydue to clever camera operations (such as moving actors about a stage tomake one appear larger than the other despite the actors actually beingof similar build, or using miniature objects with other miniatureobjects so as to be interpreted as a scene containing life-sizedobjects).

Creating video sequences for story-telling or other purposes often callsfor scenes that cannot be created with live actors, such as a talkingtree, an anthropomorphic object, space battles, and the like. Such videosequences might be generated computationally rather than capturing lightfrom live scenes. In some instances, an entirety of a video sequencemight be generated computationally, as in the case of acomputer-animated feature film. In some video sequences, it is desirableto have some computer-generated imagery and some live action, perhapswith some careful merging of the two.

While computer-generated imagery might be creatable by manuallyspecifying each color value for each pixel in each frame, this is likelytoo tedious to be practical. As a result, a creator uses various toolsto specify the imagery at a higher level. As an example, an artist mightspecify the positions in a scene space, such as a three-dimensionalcoordinate system, of objects and/or lighting, as well as a cameraviewpoint, and a camera view plane. From that, a rendering engine couldtake all of those as inputs, and compute each of the pixel color valuesin each of the frames. In another example, an artist specifies positionand movement of an articulated object having some specified texturerather than specifying the color of each pixel representing thatarticulated object in each frame.

In a specific example, a rendering engine performs ray tracing wherein apixel color value is determined by computing which objects lie along aray traced in the scene space from the camera viewpoint through a pointor portion of the camera view plane that corresponds to that pixel. Forexample, a camera view plane might be represented as a rectangle havinga position in the scene space that is divided into a grid correspondingto the pixels of the ultimate image to be generated, and if a raydefined by the camera viewpoint in the scene space and a given pixel inthat grid first intersects a solid, opaque, blue object, that givenpixel is assigned the color blue. Of course, for moderncomputer-generated imagery, determining pixel colors—and therebygenerating imagery—can be more complicated, as there are lightingissues, reflections, interpolations, and other considerations.

As illustrated in FIG. 6 , a live action capture system 602 captures alive scene that plays out on a stage 604. Live action capture system 602is described herein in greater detail, but might include computerprocessing capabilities, image processing capabilities, one or moreprocessors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown.

In a specific live action capture system, cameras 606(1) and 606(2)capture the scene, while in some systems, there might be other sensor(s)608 that capture information from the live scene (e.g., infraredcameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).On stage 604, there might be human actors, animal actors, inanimateobjects, background objects, and possibly an object such as a greenscreen 610 that is designed to be captured in a live scene recording insuch a way that it is easily overlaid with computer-generated imagery.Stage 604 might also contain objects that serve as fiducials, such asfiducials 612(1)-(3), that might be used post-capture to determine wherean object was during capture. A live action scene might be illuminatedby one or more lights, such as an overhead light 614.

During or following the capture of a live action scene, live actioncapture system 602 might output live action footage to a live actionfootage storage 620. A live action processing system 622 might processlive action footage to generate data about that live action footage andstore that data into a live action metadata storage 624. Live actionprocessing system 622 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Live action processing system 622 might processlive action footage to determine boundaries of objects in a frame ormultiple frames, determine locations of objects in a live action scene,where a camera was relative to some action, distances between movingobjects and fiducials, etc. Where elements have sensors attached to themor are detected, the metadata might include location, color, andintensity of overhead light 614, as that might be useful inpost-processing to match computer-generated lighting on objects that arecomputer-generated and overlaid on the live action footage. Live actionprocessing system 622 might operate autonomously, perhaps based onpredetermined program instructions, to generate and output the liveaction metadata upon receiving and inputting the live action footage.The live action footage can be camera-captured data as well as data fromother sensors.

An animation creation system 630 is another part of visual contentgeneration system 600. Animation creation system 630 might includecomputer processing capabilities, image processing capabilities, one ormore processors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown. Animation creationsystem 630 might be used by animation artists, managers, and others tospecify details, perhaps programmatically and/or interactively, ofimagery to be generated. From user input and data from a database orother data source, indicated as a data store 632, animation creationsystem 630 might generate and output data representing objects (e.g., ahorse, a human, a ball, a teapot, a cloud, a light source, a texture,etc.) to an object storage 634, generate and output data representing ascene into a scene description storage 636, and/or generate and outputdata representing animation sequences to an animation sequence storage638.

Scene data might indicate locations of objects and other visualelements, values of their parameters, lighting, camera location, cameraview plane, and other details that a rendering engine 650 might use torender CGI imagery. For example, scene data might include the locationsof several articulated characters, background objects, lighting, etc.specified in a two-dimensional space, three-dimensional space, or otherdimensional space (such as a 2.5-dimensional space, three-quarterdimensions, pseudo-3D spaces, etc.) along with locations of a cameraviewpoint and view place from which to render imagery. For example,scene data might indicate that there is to be a red, fuzzy, talking dogin the right half of a video and a stationary tree in the left half ofthe video, all illuminated by a bright point light source that is aboveand behind the camera viewpoint. In some cases, the camera viewpoint isnot explicit, but can be determined from a viewing frustum. In the caseof imagery that is to be rendered to a rectangular view, the frustumwould be a truncated pyramid. Other shapes for a rendered view arepossible and the camera view plane could be different for differentshapes.

Animation creation system 630 might be interactive, allowing a user toread in animation sequences, scene descriptions, object details, etc.and edit those, possibly returning them to storage to update or replaceexisting data. As an example, an operator might read in objects fromobject storage into a baking processor 642 that would transform thoseobjects into simpler forms and return those to object storage 634 as newor different objects. For example, an operator might read in an objectthat has dozens of specified parameters (movable joints, color options,textures, etc.), select some values for those parameters and then save abaked object that is a simplified object with now fixed values for thoseparameters.

Rather than requiring user specification of each detail of a scene, datafrom data store 632 might be used to drive object presentation. Forexample, if an artist is creating an animation of a spaceship passingover the surface of the Earth, instead of manually drawing or specifyinga coastline, the artist might specify that animation creation system 630is to read data from data store 632 in a file containing coordinates ofEarth coastlines and generate background elements of a scene using thatcoastline data.

Animation sequence data might be in the form of time series of data forcontrol points of an object that has attributes that are controllable.For example, an object might be a humanoid character with limbs andjoints that are movable in manners similar to typical human movements.An artist can specify an animation sequence at a high level, such as“the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) overtime T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5degrees per frame”) or even at a very high level (e.g., “character Ashould move, consistent with the laws of physics that are given for thisscene, from point P1 to point P2 along a specified path”).

Animation sequences in an animated scene might be specified by whathappens in a live action scene. An animation driver generator 644 mightread in live action metadata, such as data representing movements andpositions of body parts of a live actor during a live action scene.Animation driver generator 644 might generate corresponding animationparameters to be stored in animation sequence storage 638 for use inanimating a CGI object. This can be useful where a live action scene ofa human actor is captured while wearing mo-cap fiducials (e.g.,high-contrast markers outside actor clothing, high-visibility paint onactor skin, face, etc.) and the movement of those fiducials isdetermined by live action processing system 622. Animation drivergenerator 644 might convert that movement data into specifications ofhow joints of an articulated CGI character are to move over time.

A rendering engine 650 can read in animation sequences, scenedescriptions, and object details, as well as rendering engine controlinputs, such as a resolution selection and a set of renderingparameters. Resolution selection might be useful for an operator tocontrol a trade-off between speed of rendering and clarity of detail, asspeed might be more important than clarity for a movie maker to testsome interaction or direction, while clarity might be more importantthan speed for a movie maker to generate data that will be used forfinal prints of feature films to be distributed. Rendering engine 650might include computer processing capabilities, image processingcapabilities, one or more processors, program code storage for storingprogram instructions executable by the one or more processors, as wellas user input devices and user output devices, not all of which areshown.

Visual content generation system 600 can also include a merging system660 that merges live footage with animated content. The live footagemight be obtained and input by reading from live action footage storage620 to obtain live action footage, by reading from live action metadatastorage 624 to obtain details such as presumed segmentation in capturedimages segmenting objects in a live action scene from their background(perhaps aided by the fact that green screen 610 was part of the liveaction scene), and by obtaining CGI imagery from rendering engine 650.

A merging system 660 might also read data from rulesets formerging/combining storage 662. A very simple example of a rule in aruleset might be “obtain a full image including a two-dimensional pixelarray from live footage, obtain a full image including a two-dimensionalpixel array from rendering engine 650, and output an image where eachpixel is a corresponding pixel from rendering engine 650 when thecorresponding pixel in the live footage is a specific color of green,otherwise output a pixel value from the corresponding pixel in the livefootage.”

Merging system 660 might include computer processing capabilities, imageprocessing capabilities, one or more processors, program code storagefor storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Merging system 660 might operate autonomously,following programming instructions, or might have a user interface orprogrammatic interface over which an operator can control a mergingprocess. In some embodiments, an operator can specify parameter valuesto use in a merging process and/or might specify specific tweaks to bemade to an output of merging system 660, such as modifying boundaries ofsegmented objects, inserting blurs to smooth out imperfections, oradding other effects. Based on its inputs, merging system 660 can outputan image to be stored in a static image storage 670 and/or a sequence ofimages in the form of video to be stored in an animated/combined videostorage 672.

Thus, as described, visual content generation system 600 can be used togenerate video that combines live action with computer-generatedanimation using various components and tools, some of which aredescribed in more detail herein. While visual content generation system600 might be useful for such combinations, with suitable settings, itcan be used for outputting entirely live action footage or entirely CGIsequences. The code may also be provided and/or carried by a transitorycomputer readable medium, e.g., a transmission medium such as in theform of a signal transmitted over a network.

According to one embodiment, the techniques described herein areimplemented by one or more generalized computing systems programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Special-purpose computingdevices may be used, such as desktop computer systems, portable computersystems, handheld devices, networking devices or any other device thatincorporates hard-wired and/or program logic to implement thetechniques.

One embodiment might include a carrier medium carrying image data thatincludes image data having shadow details generated using the methodsdescribed herein. The carrier medium can comprise any medium suitablefor carrying the image data, including a storage medium, e.g.,solid-state memory, an optical disk or a magnetic disk, or a transientmedium, e.g., a signal carrying the image data such as a signaltransmitted over a network, a digital signal, a radio frequency signal,an acoustic signal, an optical signal or an electrical signal.

For example, FIG. 7 is a block diagram that illustrates a computersystem 700 upon which the computer systems of the systems describedherein and/or visual content generation system 600 (see FIG. 6 ) may beimplemented. Computer system 700 includes a bus 702 or othercommunication mechanism for communicating information, and a processor704 coupled with bus 702 for processing information. Processor 704 maybe, for example, a general-purpose microprocessor.

Computer system 700 also includes a main memory 706, such as arandom-access memory (RAM) or other dynamic storage device, coupled tobus 702 for storing information and instructions to be executed byprocessor 704. Main memory 706 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 704. Such instructions, whenstored in non-transitory storage media accessible to processor 704,render computer system 700 into a special-purpose machine that iscustomized to perform the operations specified in the instructions.

Computer system 700 further includes a read only memory (ROM) 708 orother static storage device coupled to bus 702 for storing staticinformation and instructions for processor 704. A storage device 710,such as a magnetic disk or optical disk, is provided and coupled to bus702 for storing information and instructions.

Computer system 700 may be coupled via bus 702 to a display 712, such asa computer monitor, for displaying information to a computer user. Aninput device 714, including alphanumeric and other keys, is coupled tobus 702 for communicating information and command selections toprocessor 704. Another type of user input device is a cursor control716, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor704 and for controlling cursor movement on display 712. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 700 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 700 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 700 in response to processor 704 executing one or more sequencesof one or more instructions contained in main memory 706. Suchinstructions may be read into main memory 706 from another storagemedium, such as storage device 710. Execution of the sequences ofinstructions contained in main memory 706 causes processor 704 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may includenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 710.Volatile media includes dynamic memory, such as main memory 706. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, an EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire, and fiber optics, including thewires that include bus 702. Transmission media can also take the form ofacoustic or light waves, such as those generated during radio-wave andinfra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 704 for execution. For example,the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over anetwork connection. A modem or network interface local to computersystem 700 can receive the data. Bus 702 carries the data to main memory706, from which processor 704 retrieves and executes the instructions.The instructions received by main memory 706 may optionally be stored onstorage device 710 either before or after execution by processor 704.

Computer system 700 also includes a communication interface 718 coupledto bus 702. Communication interface 718 provides a two-way datacommunication coupling to a network link 720 that is connected to alocal network 722. For example, communication interface 718 may be anetwork card, a modem, a cable modem, or a satellite modem to provide adata communication connection to a corresponding type of telephone lineor communications line. Wireless links may also be implemented. In anysuch implementation, communication interface 718 sends and receiveselectrical, electromagnetic, or optical signals that carry digital datastreams representing various types of information.

Network link 720 typically provides data communication through one ormore networks to other data devices. For example, network link 720 mayprovide a connection through local network 722 to a host computer 724 orto data equipment operated by an Internet Service Provider (ISP) 726.ISP 726 in turn provides data communication services through theworld-wide packet data communication network now commonly referred to asthe “Internet” 728. Local network 722 and Internet 728 both useelectrical, electromagnetic, or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 720 and through communication interface 718, which carrythe digital data to and from computer system 700, are example forms oftransmission media.

Computer system 700 can send messages and receive data, includingprogram code, through the network(s), network link 720, andcommunication interface 718. In the Internet example, a server 730 mighttransmit a requested code for an application program through theInternet 728, ISP 726, local network 722, and communication interface718. The received code may be executed by processor 704 as it isreceived, and/or stored in storage device 710, or other non-volatilestorage for later execution.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory. The code may also be provided carried by atransitory computer readable medium e.g., a transmission medium such asin the form of a signal transmitted over a network.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

The use of examples, or exemplary language (e.g., “such as”) providedherein, is intended merely to better illuminate embodiments of theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element as essential to thepractice of the invention.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and combinations, additions, re-arrangements, and the likeare contemplated in alternative embodiments of the present invention.Thus, while the invention has been described with respect to exemplaryembodiments, one skilled in the art will recognize that numerousmodifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

1. A computer-implemented method of processing textures to determine asample, the method comprising: under the control of one or more computersystems configured with executable instructions: receive a filter regionrepresentation, wherein the filter region representation comprisescomputer-readable data corresponding to sample region of a virtualmapping to be sampled; determining a first texture dataset; determininga filter kernel for the filter region representation, the filter kernelcorresponding to the sample region, wherein the filter kernel comprisesone or more kernel weights over the sample region; storing the filterkernel; determining a subset of the first texture dataset upon which thefilter kernel is to be applied; storing a schedule of the subset,wherein positions of texels within the first texture dataset aredeterminable from the schedule; applying the schedule and the filterkernel to the first texture dataset to determine a first texturecontribution for the sample region; determining a second texturedataset; applying the schedule and the filter kernel to the secondtexture dataset to determine a second texture contribution for thesample region; and combining the first texture contribution and thesecond texture contribution to determine an overall accumulated outputfor the sample region as the sample.
 2. The computer-implemented methodof claim 1, wherein the texture dataset comprises a plurality of levelsof varying resolution and the schedule indicates which level or levelsapply to the sample region.
 3. The computer-implemented method of claim2, wherein the plurality of levels of varying resolution comprises aMIP-map structure, a RIP-map structure, and/or a wavelet structure. 4.The computer-implemented method of claim 1, wherein the first texturedataset and/or the second texture dataset comprises a two-dimensional ora three-dimensional texture.
 5. The computer-implemented method of claim1, wherein a first subset of the first texture dataset and/or a secondsubset of the second texture dataset comprises one or more texture tilesof a predetermined size.
 6. The computer-implemented method of claim 1,wherein the sample region is computed based on a mapping of a region asurface in a virtual scene represented by geometrical models.
 7. Thecomputer-implemented method of claim 1, wherein the filter is determinedbased on a determined impact of applying the filter across a range oftile vertices.
 8. The computer-implemented method of claim 1, furthercomprising: accessing one or more texture tiles of one or more levels ofvarying resolution determined to correspond to a pixel region selectedfor filtering for the sample region; and applying at least one fittedfilter to the accessed one or more tiles.
 9. The computer-implementedmethod of claim 1, wherein the sample region is an ellipse.
 10. Thecomputer-implemented method of claim 1, wherein a fitted filter is afirst fitted filter and a plurality of fitted filters are calculated andaccumulated into a representation of the fitted filter in a cache andaccumulating the one or more tiles of one or more levels of varyingresolution to access.
 11. The computer-implemented method of claim 10,further comprising: accumulating a list of the one or more tiles toaccess for subsequent access of those one or more tiles.
 12. Thecomputer-implemented method of claim 1, further comprising generating aplurality of samples for a plurality of requests for samples and cachingdata values across requests.
 13. The computer-implemented method ofclaim 12, further comprising grouping texture dataset requests acrossthe requests.